Skip to content

Fix ETH3D ground-truth preparation and evaluation - #51

Merged
Zador-Pataki merged 1 commit into
mainfrom
zador/eth3d-ground-truth-evaluation-public
Oct 30, 2026
Merged

Fix ETH3D ground-truth preparation and evaluation#51
Zador-Pataki merged 1 commit into
mainfrom
zador/eth3d-ground-truth-evaluation-public

Conversation

@Zador-Pataki

@Zador-Pataki Zador-Pataki commented Oct 30, 2026 •
edited
Loading

Copy link
Copy Markdown
Collaborator

Motivation

Preparation assigned every image the nearest available GT pose, regardless of the time difference. Images without valid GT therefore contributed incorrect reference poses to alignment and scoring. This PR excludes those unsupported frames from evaluation alignment and scoring.

Separately, GT timestamp synchronization now follows the BAD SLAM evaluator. This has a smaller effect on the scores and follows the original implementation more closely.

Changes

  • Replace nearest-sample GT assignment with linear translation interpolation and rotation SLERP at image timestamps. This also changes reference poses for supported frames.
  • Apply the separate mocap and SfM interpolation-gap limits and retain unsupported images as unposed. Older ETH3D outputs must be prepared again.
  • Exclude unavailable GT from alignment and scoring without filling those gaps by interpolation.

ETH3D-SLAM results

Pose AUC (%), before → after; higher is better.

Uncalibrated

Absolute Pose AUC (%)

Method 5 cm 10 cm 50 cm 1 m 10 m
DA3-Long 19.03 → 21.24 37.21 → 39.78 74.14 → 75.58 84.86 → 86.14 97.98 → 98.48
GLOMAP-LG 22.35 → 26.25 33.73 → 38.26 60.59 → 64.77 68.42 → 71.23 77.92 → 78.82
GLOMAP-RoMA 28.61 → 32.77 40.28 → 44.33 57.27 → 60.38 61.20 → 63.50 65.56 → 67.06
Lingbot-Map 13.11 → 14.58 27.83 → 30.49 68.68 → 71.11 81.48 → 83.26 97.22 → 97.67
LoGeR 6.42 → 6.88 16.89 → 17.75 57.36 → 58.34 73.71 → 74.40 95.87 → 96.35
MASt3R-SLAM 8.92 → 9.83 21.58 → 23.40 65.12 → 66.15 76.83 → 77.98 96.45 → 96.56
MegaSaM 38.42 → 43.67 53.99 → 57.01 79.45 → 78.71 86.98 → 86.67 97.91 → 98.26
VGGT-SLAM2 26.15 → 30.30 43.97 → 48.23 76.61 → 78.14 84.88 → 86.05 96.99 → 97.54
ViPE 38.93 → 45.74 55.74 → 61.66 80.32 → 83.50 86.90 → 89.13 97.60 → 98.15
VidMap 51.40 → 60.07 68.30 → 75.63 90.10 → 92.85 94.00 → 95.94 99.00 → 99.44

Relative to the best score (%)

Before → after (change in percentage points). Each column is normalized to its best score separately before and after, using the values above.

Method 5 cm 10 cm 50 cm 1 m 10 m
DA3-Long 37.02 → 35.36 (−1.66) 54.48 → 52.60 (−1.88) 82.29 → 81.40 (−0.89) 90.28 → 89.79 (−0.49) 98.97 → 99.03 (+0.06)
GLOMAP-LG 43.48 → 43.70 (+0.22) 49.39 → 50.59 (+1.20) 67.25 → 69.76 (+2.51) 72.79 → 74.24 (+1.46) 78.71 → 79.26 (+0.56)
GLOMAP-RoMA 55.66 → 54.55 (−1.11) 58.98 → 58.61 (−0.36) 63.56 → 65.03 (+1.47) 65.11 → 66.19 (+1.08) 66.22 → 67.44 (+1.22)
Lingbot-Map 25.51 → 24.27 (−1.23) 40.75 → 40.31 (−0.43) 76.23 → 76.59 (+0.36) 86.68 → 86.78 (+0.10) 98.20 → 98.22 (+0.02)
LoGeR 12.49 → 11.45 (−1.04) 24.73 → 23.47 (−1.26) 63.66 → 62.83 (−0.83) 78.41 → 77.55 (−0.87) 96.84 → 96.89 (+0.05)
MASt3R-SLAM 17.35 → 16.36 (−0.99) 31.60 → 30.94 (−0.66) 72.28 → 71.24 (−1.03) 81.73 → 81.28 (−0.45) 97.42 → 97.10 (−0.32)
MegaSaM 74.75 → 72.70 (−2.05) 79.05 → 75.38 (−3.67) 88.18 → 84.77 (−3.41) 92.53 → 90.34 (−2.19) 98.90 → 98.81 (−0.09)
VGGT-SLAM2 50.88 → 50.44 (−0.43) 64.38 → 63.77 (−0.61) 85.03 → 84.16 (−0.87) 90.30 → 89.69 (−0.61) 97.97 → 98.09 (+0.12)
ViPE 75.74 → 76.14 (+0.41) 81.61 → 81.53 (−0.08) 89.15 → 89.93 (+0.78) 92.45 → 92.90 (+0.46) 98.59 → 98.70 (+0.12)
VidMap 100.00 100.00 100.00 100.00 100.00

README result update

VidMap results reported in #21 → reevaluation of the same saved trajectories with this PR (55/55 sequences):

Dataset 5 cm 10 cm 50 cm 1 m 10 m
ETH3D-SLAM 63.96 → 78.48 74.59 → 85.74 89.19 → 94.39 93.25 → 96.61 98.91 → 99.52

@Zador-Pataki
Zador-Pataki merged commit 2e919aa into main Oct 30, 2026
1 check passed
@Zador-Pataki
Zador-Pataki deleted the zador/eth3d-ground-truth-evaluation-public branch October 30, 2026 19:42
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

1 participant

@Zador-Pataki